Skip to content

Integration of Fast-LLM #2913

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

Integration of Fast-LLM #2913

wants to merge 1 commit into from

Conversation

bigximik
Copy link

Fast-LLM Integration (Draft)

This is a basic integration of Fast-LLM and is a work in progress:

  • Based on the existing HFLM integration.
  • The HFLM constructor is incompatible, so all components are initialized explicitly in our constructor.
  • Tests are adapted from sglang since neither sglang nor our implementation supports "EleutherAI/pythia-70m"; we use Qwen2 instead. However, the test coverage is not yet as complete as in the HF integration tests.
  • Quantized models are not supported in this iteration.

Still to be done:

  • Integration of distributed inference.
  • A test case for passing an instantiated Fast-LLM model?

Notes:

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Toolkit User seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@bigximik bigximik closed this Apr 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@baberabb baberabb Awaiting requested review from baberabb baberabb will be requested when the pull request is marked ready for review baberabb is a code owner

@StellaAthena StellaAthena Awaiting requested review from StellaAthena StellaAthena will be requested when the pull request is marked ready for review StellaAthena is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bigximik @CLAassistant